{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hs0YmQqsdL-C"
   },
   "source": [
    "# Spark RDD 基础\n",
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"http://nbviewer.ipython.org/github/ShowMeAI-Hub/awesome-AI-cheatsheets/blob/main/Spark/Spark_RDD_cheatsheet_code.ipynb\"><img src=\"https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg\" />在nbviewer上查看notebook</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/ShowMeAI-Hub/awesome-AI-cheatsheets/blob/main/Spark/Spark_RDD_cheatsheet_code.ipynb\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" />在Google Colab运行</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://github.com/ShowMeAI-Hub/awesome-AI-cheatsheets/tree/main/Spark/Spark_RDD_cheatsheet_code.ipynb\"><img src=\"https://badgen.net/badge/open/github/color=cyan?icon=github\" />在Github上查看源代码</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://github.com/ShowMeAI-Hub/awesome-AI-cheatsheets/Spark/Spark_RDD速查表.pdf\"><img src=\"https://badgen.net/badge/download/pdf/color=white?icon=github\"/>下载速查表</a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 说明\n",
    "**notebook by [韩信子](https://github.com/HanXinzi-AI)@[ShowMeAI](https://github.com/ShowMeAI-Hub)**\n",
    "\n",
    "更多AI速查表资料请查看[速查表大全](https://github.com/ShowMeAI-Hub/awesome-AI-cheatsheets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "J2bq0VexdL-E"
   },
   "source": [
    "PySpark是Spark的PythonAPI，允许Python调用Spark编程模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OnskyKbfdL-F"
   },
   "source": [
    "## 配置spark环境"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "RBwToZmkdL-F"
   },
   "outputs": [],
   "source": [
    "!apt-get install openjdk-8-jdk-headless -qq > /dev/null\n",
    "!wget -q www-us.apache.org/dist/spark/spark-2.4.8/spark-2.4.8-bin-hadoop2.7.tgz  \n",
    "!tar xf spark-2.4.8-bin-hadoop2.7.tgz\n",
    "!pip install -q findspark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "VEaNLWu-dL-G"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ[\"JAVA_HOME\"]=\"/usr/lib/jvm/java-8-openjdk-amd64\"\n",
    "os.environ[\"SPARK_HOME\"]=\"/content/spark-2.4.8-bin-hadoop2.7\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "MLUOaOAPdL-G"
   },
   "outputs": [],
   "source": [
    "import findspark\n",
    "findspark.init()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "t31OoLMWdL-G"
   },
   "source": [
    "## 初始化Spark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "07Jd6sMRdL-H"
   },
   "source": [
    "### SparkContext"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "jAEh8ndodL-H"
   },
   "outputs": [],
   "source": [
    "from pyspark import SparkContext"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "YVlZvQEmdL-H"
   },
   "outputs": [],
   "source": [
    "sc = SparkContext(master = 'local[2]')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mFWHqPv5dL-I"
   },
   "source": [
    "### SparkContext信息获取"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "laKMJk1mdL-I",
    "outputId": "8500b7fe-5aec-458b-d4d3-f9f90c14db0d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.4.8'"
      ]
     },
     "execution_count": 6,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sc.version   #获取SparkContext版本"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "ygvEDogzdL-I",
    "outputId": "59e068e9-5c41-436e-b6f9-34fd7087008b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'3.7'"
      ]
     },
     "execution_count": 7,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sc.pythonVer   #获取Python版本"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "koTBXPH0dL-J",
    "outputId": "cd6b735f-ee4d-44e1-e752-929868b7efe4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'local[2]'"
      ]
     },
     "execution_count": 8,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sc.master   #要连接的MasterURL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "cxbb0EMUdL-J",
    "outputId": "4b103f4c-82b4-4dde-be93-b1f2cc1fcf93"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'None'"
      ]
     },
     "execution_count": 9,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "str(sc.sparkHome)   #Spark在工作节点的安装路径"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "_gbG9HfIdL-J",
    "outputId": "28bad3ab-1ae3-4852-c1d1-b83dd163112f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'root'"
      ]
     },
     "execution_count": 10,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "str(sc.sparkUser())   #获取SparkContext的Spark用户名"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "GvE8OWPmdL-J",
    "outputId": "4b1de164-c54a-4fe7-d698-25d9b6b6c874"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'pyspark-shell'"
      ]
     },
     "execution_count": 11,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sc.appName   #返回应用名称"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "S-9gAoAbdL-J",
    "outputId": "50d8c685-907e-4e35-8b07-41ba2bfb0079"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'local-1623220140497'"
      ]
     },
     "execution_count": 12,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sc.applicationId   #获取应用程序ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "bhmmz2SldL-K",
    "outputId": "5d98ba64-6d8d-4516-b7ab-fd0f4abfe55a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 13,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sc.defaultParallelism   #返回默认并行级别"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ITySbg_ndL-K",
    "outputId": "11abf19e-811c-4a48-af15-2e1cee00c3d5"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 14,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sc.defaultMinPartitions   #RDD默认最小分区数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kkR3lLXidL-K"
   },
   "source": [
    "### 配置"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "NAJ0FMxmdL-K"
   },
   "outputs": [],
   "source": [
    "from pyspark import SparkConf, SparkContext\n",
    "\n",
    "conf = (SparkConf() \\\n",
    "        .setMaster(\"local\") \\\n",
    "        .setAppName(\"My app\") \\\n",
    "        .set(\"spark.executor.memory\", \"1g\"))\n",
    "\n",
    "sc = SparkContext.getOrCreate(conf = conf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cbdfKYomdL-K"
   },
   "source": [
    "### 使用Shell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NEEzOqt8dL-L"
   },
   "source": [
    "PySpark Shell已经为SparkContext创建了名为 sc 的变量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jg-_jCsbdL-L"
   },
   "outputs": [],
   "source": [
    "$./bin/spark-shell --master local[2]        #命令行启动spark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "T0AF3mWQdL-L"
   },
   "outputs": [],
   "source": [
    "$./bin/pyspark --master local[4] --py-files code.py     #命令行提交spark脚本任务"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_yrvtU2OdL-L"
   },
   "source": [
    "用 --master 参数设定 Context 连接到哪个Master 务器，通过传递逗号分隔列表至 --py-files 添加 Python.zip、.egg 或 .py文件到 Runtime 路径。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wQdR4SwsdL-L"
   },
   "source": [
    "## 加载数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QWzCaZ4zdL-M"
   },
   "source": [
    "### 并行集合"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "6CBxyI7ndL-M"
   },
   "outputs": [],
   "source": [
    "rdd = sc.parallelize([('a',7),('a',2),('b',2)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "qPxn-HxLdL-M"
   },
   "outputs": [],
   "source": [
    "rdd2 = sc.parallelize([('a',2),('d',1),('b',1)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "txGjsAyDdL-M"
   },
   "outputs": [],
   "source": [
    "rdd3 = sc.parallelize(range(100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "8l0JMsROdL-M"
   },
   "outputs": [],
   "source": [
    "rdd4 = sc.parallelize([(\"a\",[\"x\",\"y\",\"z\"]), (\"b\",[\"p\", \"r\"])])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uFRgDGqXdL-M"
   },
   "source": [
    "### 外部数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MFn2DFLMdL-N"
   },
   "source": [
    "使用textFile()函数从HDFS、本地文件或其它支持Hadoop的文件系统里读取文本文件，或使用wholeTextFiles()函数读取目录里文本文件。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "NtrTwpJ0dL-N"
   },
   "outputs": [],
   "source": [
    "textFile = sc.textFile(\"/my/directory/*.txt\")\n",
    "# 如果是在google colab中可以运行下方代码\n",
    "# textFile = sc.textFile(\"sample_data/california_housing_train.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4SJXrnvKdL-N"
   },
   "outputs": [],
   "source": [
    "textFile2 = sc.wholeTextFiles(\"/my/directory/\")\n",
    "# 如果是在google colab中可以运行下方代码\n",
    "# textFile2 = sc.wholeTextFiles(\"sample_data/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yosf_y1bdL-N"
   },
   "source": [
    "## 提取RDD信息"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aD5uyF3AdL-N"
   },
   "source": [
    "### 基础信息"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HbgY-H_bdL-N",
    "outputId": "4b8e92a0-a187-4f94-b2f4-3d3d67b7e220"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 21,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd.getNumPartitions()   #列出分区数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "qlImdMatdL-O",
    "outputId": "d9efb42b-de96-461c-af77-269f04e87612"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 22,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd.count()   #计算RDD实例数量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ka8HSn-ZdL-O",
    "outputId": "fa1c80c0-5a0c-427b-85cd-8d09a5050972"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "defaultdict(int, {'a': 2, 'b': 1})"
      ]
     },
     "execution_count": 23,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd.countByKey()   #按键计算RDD实例数量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ZCkzQ2lgdL-O",
    "outputId": "9bfd78a6-cd23-43f9-9aaf-3ced977d7151"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "defaultdict(int, {('a', 2): 1, ('a', 7): 1, ('b', 2): 1})"
      ]
     },
     "execution_count": 24,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd.countByValue()   #按值计算RDD实例数量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "haQ2vbSwdL-O",
    "outputId": "c5e19da5-c660-4450-a345-a359e60939b1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'a': 2, 'b': 2}"
      ]
     },
     "execution_count": 25,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd.collectAsMap()   #以字典形式返回键值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "FHJmNZ5ddL-O",
    "outputId": "77d7db55-9b52-4001-b616-085eecb2308c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4950"
      ]
     },
     "execution_count": 26,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd3.sum()   #RDD元素求和"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "TahLcknxdL-O",
    "outputId": "fae8e44a-795e-4a31-e544-1958d2689e2c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 27,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sc.parallelize([]).isEmpty()   #检查RDD是否为空"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bcjhCLrFdL-P"
   },
   "source": [
    "### 汇总"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "0f4wMs54dL-P",
    "outputId": "eb162761-8251-495f-f368-b52ff8a0c682"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "99"
      ]
     },
     "execution_count": 28,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd3.max()   #RDD元素的最大值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1g7Y7QHCdL-P",
    "outputId": "fed439f4-df1f-41f1-f6b6-73ba1eac6d1d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 29,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd3.min()   #RDD元素的最小值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "e7gsVtLTdL-P",
    "outputId": "0f166d3d-e6d2-4dfc-f969-538535df7708"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "49.5"
      ]
     },
     "execution_count": 30,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd3.mean()   #RDD元素的平均值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "fYGVjvLRdL-P",
    "outputId": "20968162-a2a0-4b7e-ffee-c1ad00a23d94"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28.86607004772212"
      ]
     },
     "execution_count": 31,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd3.stdev()   #RDD元素的标准差"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Idjsmr7JdL-P",
    "outputId": "62828a68-34a4-43e8-c0ec-38b9811fdc5e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "833.25"
      ]
     },
     "execution_count": 32,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd3.variance()   #RDD元素的方差"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "26SMhY6hdL-Q",
    "outputId": "6ef71d44-64bc-4f43-82be-c6493f3757c2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([0, 33, 66, 99], [33, 33, 34])"
      ]
     },
     "execution_count": 33,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd3.histogram(3)   #分箱(Bin)生成直方图"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "NwxGo6CPdL-Q",
    "outputId": "ebdfe4ff-6563-44cf-b35e-0a6a4cec72d6"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(count: 100, mean: 49.5, stdev: 28.86607004772212, max: 99.0, min: 0.0)"
      ]
     },
     "execution_count": 34,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd3.stats()   #综合统计包括：计数、平均值、标准差、最大值和最小值"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tLI2MwAvdL-Q"
   },
   "source": [
    "## 应用函数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KujpWl_edL-Q"
   },
   "source": [
    "**map与flatmap函数**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "p9xVZwncdL-Q",
    "outputId": "1e754bd2-4eec-4635-bd00-f511d8e94af9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('a', 7, 7, 'a'), ('a', 2, 2, 'a'), ('b', 2, 2, 'b')]"
      ]
     },
     "execution_count": 35,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd.map(lambda x: x+(x[1],x[0])).collect()   #对每个RDD元素执行函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "id": "QQi1tg_rdL-Q"
   },
   "outputs": [],
   "source": [
    "rdd5=rdd.flatMap(lambda x: x+(x[1],x[0]))   #对每个RDD元素执行函数，并拉平结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "oPIDn0v3dL-R",
    "outputId": "80eba73e-e2ac-45d2-8fec-59270304984f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a', 7, 7, 'a', 'a', 2, 2, 'a', 'b', 2, 2, 'b']"
      ]
     },
     "execution_count": 37,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd5.collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-HcWF0DAdL-R",
    "outputId": "22e2857c-b8f5-48bd-b575-8b19bc80952c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('a', 'x'), ('a', 'y'), ('a', 'z'), ('b', 'p'), ('b', 'r')]"
      ]
     },
     "execution_count": 38,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd4.flatMapValues(lambda x: x).collect()   #不改变键，对rdd4的每个键值对执行flatMap函数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "I1h6AJBcdL-R"
   },
   "source": [
    "## 选择数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "is97QgiEdL-R"
   },
   "source": [
    "### **获取**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "TTP6WDQxdL-R",
    "outputId": "d55b04d8-026d-4efe-bd3e-e26b679448ba"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('a', 7), ('a', 2), ('b', 2)]"
      ]
     },
     "execution_count": 39,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd.collect()   #返回包含所有RDD元素的列表"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "XxBi9lRldL-R",
    "outputId": "3cb6b1d5-38bc-4bfd-ec25-ef7625ffd2be"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('a', 7), ('a', 2)]"
      ]
     },
     "execution_count": 40,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd.filter(lambda x: \"a\" in x) .collect()   #提取前两个RDD元素"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4e6Qw-ZQdL-S",
    "outputId": "6136c5cd-983c-44e5-8ef9-98c14b223b22"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('a', 7)"
      ]
     },
     "execution_count": 41,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd.first()   #提取第一个RDD元素"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "OWCZgWjUdL-S",
    "outputId": "9bd2bf4c-8bc6-40a2-dbda-28eaabb710fe"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2, 'b', 'a', 7]"
      ]
     },
     "execution_count": 42,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd5.distinct().collect()   #提取前两个RDD元素"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iNaNKWJLdL-S"
   },
   "source": [
    "### 抽样"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "VgXsI3nYdL-S",
    "outputId": "74c2859f-9209-48c5-81ef-7850c3e0e516"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[3, 4, 27, 28, 35, 41, 43, 49, 51, 55, 64, 65, 66, 67, 85, 88, 89, 92]"
      ]
     },
     "execution_count": 43,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd3.sample(False, 0.15, 81).collect()   #返回rdd3的采样子集"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rLCLpAOXdL-S"
   },
   "source": [
    "### 筛选"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "wbTnzMLDdL-S",
    "outputId": "4fc111f1-d54f-4056-8612-ab15511d7c9c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('a', 7), ('a', 2)]"
      ]
     },
     "execution_count": 44,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    " rdd.filter(lambda x: \"a\" in x) .collect()   #筛选RDD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "mioOX8gwdL-S",
    "outputId": "94e510fd-fe5c-4f0f-c361-3927ea59f587"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2, 'b', 'a', 7]"
      ]
     },
     "execution_count": 45,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd5.distinct().collect()   #返回RDD里的唯一值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "KFwYVuW0dL-T",
    "outputId": "127f80f4-1fb4-4424-a01b-45c5784d01da"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a', 'a', 'b']"
      ]
     },
     "execution_count": 46,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd.keys().collect()   #返回RDD键值对里的键"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8WmMaWutdL-T"
   },
   "source": [
    "## 迭代"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2nZI-fp-dL-T"
   },
   "source": [
    "**foreach函数迭代**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "id": "Ew_BAvRedL-T"
   },
   "outputs": [],
   "source": [
    "def g(x):\n",
    "    print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "id": "BYiR_VBudL-T"
   },
   "outputs": [],
   "source": [
    "rdd.foreach(g)   #为所有RDD应用函数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "S23CSopMdL-T"
   },
   "source": [
    "## 改变数据形状"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NcJX1GYMdL-U"
   },
   "source": [
    "### Reduce操作"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "2XM_wt6zdL-U",
    "outputId": "b2c0eafe-f832-4bdd-c6c6-69fd9ec6beed"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('b', 2), ('a', 9)]"
      ]
     },
     "execution_count": 49,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd.reduceByKey(lambda x,y : x+y).collect()   #合并每个键的RDD值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "lqM1a0LhdL-U",
    "outputId": "4cd4a983-6a43-4271-9952-9e475b813fd9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('a', 7, 'a', 2, 'b', 2)"
      ]
     },
     "execution_count": 50,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd.reduce(lambda a, b: a + b)   #合并RDD的值"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "X_EImZRsdL-U"
   },
   "source": [
    "### 分组"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "C84fmP6JdL-U",
    "outputId": "eefa8777-88d6-46fa-82e9-d93d23104c8e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  [0,\n",
       "   2,\n",
       "   4,\n",
       "   6,\n",
       "   8,\n",
       "   10,\n",
       "   12,\n",
       "   14,\n",
       "   16,\n",
       "   18,\n",
       "   20,\n",
       "   22,\n",
       "   24,\n",
       "   26,\n",
       "   28,\n",
       "   30,\n",
       "   32,\n",
       "   34,\n",
       "   36,\n",
       "   38,\n",
       "   40,\n",
       "   42,\n",
       "   44,\n",
       "   46,\n",
       "   48,\n",
       "   50,\n",
       "   52,\n",
       "   54,\n",
       "   56,\n",
       "   58,\n",
       "   60,\n",
       "   62,\n",
       "   64,\n",
       "   66,\n",
       "   68,\n",
       "   70,\n",
       "   72,\n",
       "   74,\n",
       "   76,\n",
       "   78,\n",
       "   80,\n",
       "   82,\n",
       "   84,\n",
       "   86,\n",
       "   88,\n",
       "   90,\n",
       "   92,\n",
       "   94,\n",
       "   96,\n",
       "   98]),\n",
       " (1,\n",
       "  [1,\n",
       "   3,\n",
       "   5,\n",
       "   7,\n",
       "   9,\n",
       "   11,\n",
       "   13,\n",
       "   15,\n",
       "   17,\n",
       "   19,\n",
       "   21,\n",
       "   23,\n",
       "   25,\n",
       "   27,\n",
       "   29,\n",
       "   31,\n",
       "   33,\n",
       "   35,\n",
       "   37,\n",
       "   39,\n",
       "   41,\n",
       "   43,\n",
       "   45,\n",
       "   47,\n",
       "   49,\n",
       "   51,\n",
       "   53,\n",
       "   55,\n",
       "   57,\n",
       "   59,\n",
       "   61,\n",
       "   63,\n",
       "   65,\n",
       "   67,\n",
       "   69,\n",
       "   71,\n",
       "   73,\n",
       "   75,\n",
       "   77,\n",
       "   79,\n",
       "   81,\n",
       "   83,\n",
       "   85,\n",
       "   87,\n",
       "   89,\n",
       "   91,\n",
       "   93,\n",
       "   95,\n",
       "   97,\n",
       "   99])]"
      ]
     },
     "execution_count": 51,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd3.groupBy(lambda x: x % 2).mapValues(list).collect()   #返回RDD的分组值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "rQFizMCgdL-U",
    "outputId": "147a4e77-3d8f-4582-9a34-46fa4361061e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('b', [2]), ('a', [7, 2])]"
      ]
     },
     "execution_count": 52,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd.groupByKey().mapValues(list).collect()   #按键分组RDD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NsVcp_IMdL-U"
   },
   "source": [
    "### 聚合"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "id": "KPkDCeKJdL-V"
   },
   "outputs": [],
   "source": [
    "seqOp = (lambda x,y: (x[0]+y,x[1]+1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "id": "XZh-RAepdL-V"
   },
   "outputs": [],
   "source": [
    "combOp = (lambda x,y:(x[0]+y[0],x[1]+y[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "id": "J3RhS8OdfdFq"
   },
   "outputs": [],
   "source": [
    "add = (lambda x,y:x+y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Y9JGfBQmdL-V",
    "outputId": "0fb3a28b-a708-47c4-a566-e53bbb47ed7a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4950, 100)"
      ]
     },
     "execution_count": 55,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd3.aggregate((0,0), seqOp, combOp)   #汇总每个分区里的RDD元素，并输出结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "JzhsvX9UdL-V",
    "outputId": "16c46148-5e19-40bb-b2cf-67c4eb72b867"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('b', (2, 1)), ('a', (9, 2))]"
      ]
     },
     "execution_count": 60,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd.aggregateByKey((0,0), seqOp, combOp).collect()   #汇总每个RDD的键的值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Nx0ydqiudL-V",
    "outputId": "72e108ab-0683-40e0-aa03-d27c86fcfc8c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4950"
      ]
     },
     "execution_count": 67,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd3.fold(0, add)   #汇总每个分区里的RDD元素，并输出结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "RHuRrKFOdL-W",
    "outputId": "13b3a0db-ac7f-4e5c-e29b-971900e7e88f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('b', 2), ('a', 9)]"
      ]
     },
     "execution_count": 68,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd.foldByKey(0, add).collect()   #合并每个键的值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "3bdb_RuSdL-W",
    "outputId": "45125017-64c5-4738-f4d9-8bd442e863ff"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0, 0),\n",
       " (2, 1),\n",
       " (4, 2),\n",
       " (6, 3),\n",
       " (8, 4),\n",
       " (10, 5),\n",
       " (12, 6),\n",
       " (14, 7),\n",
       " (16, 8),\n",
       " (18, 9),\n",
       " (20, 10),\n",
       " (22, 11),\n",
       " (24, 12),\n",
       " (26, 13),\n",
       " (28, 14),\n",
       " (30, 15),\n",
       " (32, 16),\n",
       " (34, 17),\n",
       " (36, 18),\n",
       " (38, 19),\n",
       " (40, 20),\n",
       " (42, 21),\n",
       " (44, 22),\n",
       " (46, 23),\n",
       " (48, 24),\n",
       " (50, 25),\n",
       " (52, 26),\n",
       " (54, 27),\n",
       " (56, 28),\n",
       " (58, 29),\n",
       " (60, 30),\n",
       " (62, 31),\n",
       " (64, 32),\n",
       " (66, 33),\n",
       " (68, 34),\n",
       " (70, 35),\n",
       " (72, 36),\n",
       " (74, 37),\n",
       " (76, 38),\n",
       " (78, 39),\n",
       " (80, 40),\n",
       " (82, 41),\n",
       " (84, 42),\n",
       " (86, 43),\n",
       " (88, 44),\n",
       " (90, 45),\n",
       " (92, 46),\n",
       " (94, 47),\n",
       " (96, 48),\n",
       " (98, 49),\n",
       " (100, 50),\n",
       " (102, 51),\n",
       " (104, 52),\n",
       " (106, 53),\n",
       " (108, 54),\n",
       " (110, 55),\n",
       " (112, 56),\n",
       " (114, 57),\n",
       " (116, 58),\n",
       " (118, 59),\n",
       " (120, 60),\n",
       " (122, 61),\n",
       " (124, 62),\n",
       " (126, 63),\n",
       " (128, 64),\n",
       " (130, 65),\n",
       " (132, 66),\n",
       " (134, 67),\n",
       " (136, 68),\n",
       " (138, 69),\n",
       " (140, 70),\n",
       " (142, 71),\n",
       " (144, 72),\n",
       " (146, 73),\n",
       " (148, 74),\n",
       " (150, 75),\n",
       " (152, 76),\n",
       " (154, 77),\n",
       " (156, 78),\n",
       " (158, 79),\n",
       " (160, 80),\n",
       " (162, 81),\n",
       " (164, 82),\n",
       " (166, 83),\n",
       " (168, 84),\n",
       " (170, 85),\n",
       " (172, 86),\n",
       " (174, 87),\n",
       " (176, 88),\n",
       " (178, 89),\n",
       " (180, 90),\n",
       " (182, 91),\n",
       " (184, 92),\n",
       " (186, 93),\n",
       " (188, 94),\n",
       " (190, 95),\n",
       " (192, 96),\n",
       " (194, 97),\n",
       " (196, 98),\n",
       " (198, 99)]"
      ]
     },
     "execution_count": 59,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd3.keyBy(lambda x: x+x).collect()   #通过执行函数，创建RDD元素的元组"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zwqoh0GbdL-W"
   },
   "source": [
    "## 数学运算"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0pAj2aA4dL-W"
   },
   "source": [
    "**RDD运算**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5dlbr6uGdL-W",
    "outputId": "2fab984e-3a7a-4c62-81b6-faa03a1b90b2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('b', 2), ('a', 7)]"
      ]
     },
     "execution_count": 69,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd.subtract(rdd2).collect()   #返回在rdd2里没有匹配键的rdd键值对"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4-oyLcaHdL-W",
    "outputId": "b999da59-2f76-4e23-c45b-048d5987e2b0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('d', 1)]"
      ]
     },
     "execution_count": 70,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd2.subtractByKey(rdd).collect()   #返回rdd2里的每个(键，值)对，rdd中没有匹配的键"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-zzr0AvpdL-W",
    "outputId": "58f2a66c-3465-4f0e-f5ca-9e4dd85171e0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(('a', 7), ('a', 2)),\n",
       " (('a', 7), ('d', 1)),\n",
       " (('a', 7), ('b', 1)),\n",
       " (('a', 2), ('a', 2)),\n",
       " (('b', 2), ('a', 2)),\n",
       " (('a', 2), ('d', 1)),\n",
       " (('a', 2), ('b', 1)),\n",
       " (('b', 2), ('d', 1)),\n",
       " (('b', 2), ('b', 1))]"
      ]
     },
     "execution_count": 71,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd.cartesian(rdd2).collect()   #返回rdd和rdd2的笛卡尔积"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "x0mVRHHRdL-X"
   },
   "source": [
    "## 排序"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d2QFc2ajdL-X"
   },
   "source": [
    "**RDD排序**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ercn6T4mdL-X",
    "outputId": "24f14bdb-a9e8-4128-d989-7d018cf9f2e9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('d', 1), ('b', 1), ('a', 2)]"
      ]
     },
     "execution_count": 72,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd2.sortBy(lambda x: x[1]).collect()   #按给定函数排序"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "QhdWO-LadL-X",
    "outputId": "b74fe886-8dbc-487c-eadf-5b7e4104c7d2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('a', 2), ('b', 1), ('d', 1)]"
      ]
     },
     "execution_count": 73,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd2.sortByKey() .collect()   #RDD按键排序RDD的键值对"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZJtow4W3dL-X"
   },
   "source": [
    "## 重分区"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PSfgRO9YdL-X"
   },
   "source": [
    "**repartition函数**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "xRGe2WBfdL-X",
    "outputId": "0480f54b-69b8-465c-ee49-ddb13dd17274"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MapPartitionsRDD[104] at coalesce at NativeMethodAccessorImpl.java:0"
      ]
     },
     "execution_count": 74,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd.repartition(4)   #新建一个含4个分区的RDD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "bBgo3eJgdL-Y",
    "outputId": "07725939-0985-4d86-f018-d44bd9a446cc"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CoalescedRDD[105] at coalesce at NativeMethodAccessorImpl.java:0"
      ]
     },
     "execution_count": 75,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd.coalesce(1)   #将RDD中的分区数缩减为1个"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OzCqO22GdL-Y"
   },
   "source": [
    "## 保存"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "l7iin8EqdL-Y"
   },
   "source": [
    "**存储RDD到本地或HDFS**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "id": "whs6HkLqdL-Y"
   },
   "outputs": [],
   "source": [
    "rdd.saveAsTextFile(\"rdd.txt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "OMkkjXCQdL-Y"
   },
   "outputs": [],
   "source": [
    "rdd.saveAsHadoopFile(\"hdfs://namenodehost/parent/child\", 'org.apache.hadoop.mapred.TextOutputFormat')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "j12wsL6jdL-Y"
   },
   "source": [
    "## 终止SparkContext"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Cxvc8rXNdL-Z"
   },
   "source": [
    "**停止SparkContext**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "id": "S2ax2CbWdL-Z"
   },
   "outputs": [],
   "source": [
    "sc.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "m1yaq3ygdL-Z"
   },
   "source": [
    "## 执行脚本程序"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MyW24J2edL-Z"
   },
   "source": [
    "**提交脚本执行**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_Mb3my13dL-Z"
   },
   "outputs": [],
   "source": [
    "$./bin/spark-submit examples/src/main/python/pi.py"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Spark-RDD-cheatsheet-code.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
